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FIELD OF THE INVENTION 

^i^^Tinvention relates to a method and a system 
for broadcasting images of moving objects, especially video 
images, with insertion of advertisement pictures or the like on 
specific locations of the moving objects. 
BACKGROUND OF THE I NVENTION 

J^Hncr changing the content of advertisement 
pictures of broadcasted views are already known. For example, 
virtual reality systems superimpose adverts on predetermxned 
static locations of a filmed area (for example, panels of a 
soccer or football stadium) . Such systems are used, for adver- 
tising purposes, to modify the content of the advertisement 
depending on, for example, the country or the geographic area in 
which the images are broadcasted. The choice of the advert xse- 
me nt can be made in view of customer habits or legal aspects 

(for example, in some countries, tobacco or alcohol advertising 

is illegal) . 

Figure 1 is a block diagram of a known method for 
superimposing panels on fixed locations of filmed areas. 

A camera has a reference position that is recorded by 
a computer, which mixes the fixed advertisement, and the televi- 
sion picture. The fact that the camera has pan and tilt move- 



merits within a certain area also means that the computer knows 
the static environment by means of pre -registration. When the 
advertisement is to be imposed on, say, the middle circle of a 
football pitch, it is inserted on the static part of the TV 
5 picture that is visible. This is done to avoid putting the 
advertisement on the players, or on any other objects not 
belonging to the static environment. 

The pre -registration phase of the static environment 
is shown to the left of the .vertical dashed line in Figure 1. 

10 'Using the football pitch as an example, cameras are panned and 
tilted so that all parts of the football pitch environment that 
can be captured (block 40) using camera lenses (blocks 41, 42, 
43) are stored in a database 44 for use in the later broadcast. 
The advertisement boundaries are then indexed in the database 

15 representation of the football pitch for every position for all 
cameras. " ■ 

Since the cameras move (pan and tilt) only within the 
boundary done in the registration phase, and the static environ- 
ment is known, the advertisements can be inserted into the 

2 0 production stream. In Figure 1, this is shown as being inserted 
after the broadcast stream has been produced (block 45) . Only 
the parts of the broadcast stream that are identical to the 
static representation of the football pitch are candidates for 
advertisement inserts (block 46) . The positioning of the adverts 

2 5 are static with respect to the football pitch, and pixel 

boundaries in the representation lock the advertising image to a 
position on the pitch rather than the camera view. 

The limitation of such systems is that the advertise- 
ments are fixed to certain points or areas in the camera views. 

3 0 It would be desirable to be able to superimpose 

pictures (especially advertisement) not only on static areas 
(eventually filmed by moving cameras) but also on moving 
objects, for example, on cars in car races. 

A first difficulty is that the main subject of the 
3 5 images (the subject followed by the camera) is the moving object 
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on which one wants to add advertisements . On the contrary, in a 
stadium, the main subject is/are the player (s) and the panels of 
the stadium on which adverts are superimposed constitute a 
background. Further, the background is often larger than the 
5 subject, at least when an advertisement panel has to be under- 
standable for the person watching. So, it is expected that the 
computation amount for tracking a moving object and superimpos- 
ing an advertisement will be greater than for superimposing an 
advertisement to a large fixed background panel. 

10 A second difficulty is that both the camera filming 

the object and the filmed object (for example a car) can move. 
This constitutes an additional factor increasing the amount of 
computation. In particular, in a system dedicated to static 
panels (for example, a stadium) , the cameras are often fixed and 

15 just move in orientation. Further, only the roll and pitch and 
not the yaw can vary, which reduces- the possible deformations of 
the predetermined pictures to be superimposed to the image of a 
fixed panel. Treating pictures provided by moving cameras (for 
example, located in a helicopter to film a car race) necessi- 

2 0 tates dealing with six degrees of freedom (three rectangular 
coordinates plus roll, pitch and yaw) . 

There is a technical problem in providing a method and 
a system for broadcasting moving images of moving objects taken 
by moving cameras and to superimpose various advertisement 

2 5 ' pictures on the filmed objects. 

There is another technical problem for making the 
required computations in real time while broadcasting the views 
(for example, for a TV program in live) . 
SUMMARY OF THE INVENTION 

3 0 A purpose of the invention is to provide a method and 

a system for adding or superimposing predetermined pictures, 
especially advertisement pictures, on predetermined location (s) 
of moving object (s) in a video stream. More generally, the 
invention aims at super imposing, in a video stream, at least one 
3 5 predetermined image portion at a predetermined location of at 
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least one moving object image, to simulate that the object 
carries the predetermined image portion on. 

Another purpose of the present invention is to provide 
a real time image processing in a live broadcasting program. 
5 Another purpose of the invention is to mix images 

taken by several moving cameras. 

To attain these purposes and others, the present 
invention provides a method and a system for superimposing, in a 
video stream, an image portion representing a picture to be 
10 ' carried by an image of a moving object in which the image acqui- 
sition is disassociated from the insertion of the image portion. 

According to the invention, at the beginning of a 
session (a TV program) , a library of the pictures in various 
orientations is downloaded (broadcasted) in the video receiver 
15 (for example, the set top box of a TV set or the like) . Then, 
along with the successive video" images, only an identifying 
reference of the picture with a size and location indication is 
broadcasted, the super imposition being made on the receiver 
side . 

2 0 According to a preferred embodiment of the invention, 

a geometrical representation of the moving object 'to which the 
system has to superimpose pictures is used so that the pictures 
can be correctly inserted in relation to a reference point on 
the object. This reference point is, according to the invention, 

2 5 the point of the object from which the position and orientation 

parameters are calculated. The position (location) of the object 
is preferably obtained by means of a conventional positioning 
system (for example, a GPS) . The moving objects to be tracked by 
the system are identified along with their location. The orien- 

3 0 tation parameters of the object are preferably obtained with a 

conventional yaw, pitch and roll sensor attached to the refer- 
ence point. 

Using a reference point minimizes the required compu- 
tation amount. Indeed, unless the object is deformed between two 
3 5 images (either because its orientation has changed or because 
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the camera has being displaced) , the position of the advertise- 
ment picture is constant in relation to the reference point for 
the car in the geometrical model . 

Hence, according to a preferred embodiment, the deter- 
5 ruination of a location, orientation and size of an object in the 
current view is, if that object was already present in the 
former view, a differential determination. 

The detection of the apparition of an object in a 
current view can be made by several ways. 
10 According to a first embodiment, not only the moving 

objects to which superimposed pictures are localizable and iden- 
tifiable by the system, but also static points of the filmed 
event place (for example, the race circuit) . Hence, knowing the 
location of the moving objects, the static points and the camera 
15 taking the current image, the system detects, along with the 
size of the picture (depending oh the zooming parameter of the 
camera) when a new object enters the field of the camera. 

According to another embodiment, the system uses a 
shape recognition tool to detect the moving objects (for example 
2 0 the cars) on the basis on their geometrical representation to 
detect the presence of such objects in the current image. 
B RIEF DESCRIPTION OF THE DRAWINGS 

The foregoing and others purposes, features, aspects 
and advantages of the invention will become apparent from the 

2 5 following detailed description of embodiments, given by way of 

illustration and not limitation with reference to the accompany- 
ing drawings. 

Figure 1 already disclosed illustrates, very schemati- 
cally, a conventional method for superimposing fixed panel on 

3 0 static locations of filmed areas ; 

Figure 2 is a block diagram of an embodiment of a 
system for superimposing pictures to images of moving objects; 

Figure 3 is a flowchart of an embodiment of the method 
for operating the system of figure 2; and 
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Figure 4 is a block diagram illustrating an embodiment 
of the method according to the present invention. 

Only the elements and steps useful to the understand- 
ing of the present invention have been shown in the drawings and 
5 will be disclosed hereafter. Especially, details of localizing 
the elements in a same referential are not disclosed. Further, 
details for calculating picture deformation using conventional 
picture processing tools and methods have not been disclosed as 
it will readily occur to one skilled in the art. 
1 0 ' D ETAILED DESCRIPTION OF THE INVENTION 

Figure 2 represents, very schematically, a system 
according to an embodiment of the present invention dedicated to 
film a car race. 

At least one source of moving images films a race 
15 circuit RC and more specifically the cars C running on the 
circuit. In the example, three video cameras 1, 2 and 3 are 
respectively attached to a helicopter 4, a truck 5 and a pole 6. 
The image streams respectively taken by the cameras are, like in 
a conventional system, transmitted to a video production mixer 7 
2 0 (VPM) either through wired (not shown) or wireless (antenna A) 
links, for example, through a satellite 8 (SAT) . Each camera, 
more precisely each camera support (helicopter, truck, pole, 
etc . ) is provided with a positioning system, for example a GPS 
or any other conventional system and, according to the inven- 

2 5 tion, also with an orientation sensor. 

The production mixer 7 transmits the mixed stream of 
images to at least one signal distribution point 9 (SDP) which 
distributes the image stream to several area distribution points 
10 (ADP1, ADP2) . In a conventional system, the signal distribu- 

3 0 tion point or the area distributions points insert fixed adver- 

tisement pictures in each image (for example, panels of a 
stadium) before transmission (for example, broadcasting) . 

According to the present invention, the signal distri- 
bution point dispatch the image stream to several area distribu- 
3 5 tion points 10 which transmit the images without modification to 



TV sets or the like, more precisely to Set Top Boxes 11 (STB^, 
STB^) of TV sets wire or wireless connected to the distribution 
points. According to the invention, the distribution points are 
not compulsorily area distribution points but can also be 
5 subject distribution points (for example, for thematic 
channels) . 

The operation of a system according to the present 
invention will be disclosed in connection with figure 3 which 
illustrates an embodiment of the method for superimposing 

10 'pictures to moving images according to the invention. 

Before broadcasting an image stream relating to an 
event (for example, a car race) to which advertisement pictures 
have to be added, advertisement pictures in various orientations 
are calculated (block 20, ADVERT PICTURES), taking into account 

15 their content and their location on the moving object of the 
scene (for example, a car) . Preferably, the orientation is 
determined on the basis of a geometrical representation of the 
moving object with respect to a reference point of this object. 
At the end of this step, a library of advertisement views in 

2 0 various orientations is constituted. Each oriented picture view 
is stored along with its location from the reference point of 
the object and with an identifying reference. 

Even if the speed of the moving object is very high 
(for example for a race car) , since the repetition frequency of 

2 5 television frame is only 25 frames per second, the number of 

oriented picture views to be stored and to be calculated is 
limited. However, for determining the number of picture views to 
be calculated for each advertisement, one could take into 
account the range of speed of the moving object with respect to 

3 0 the image frequency 

It should be noted that, even if the content of the 
advertisement varies, the shape and size of the picture on the 
moving object is often the same from an event to another. Hence, 
the above calculation based on a geometrical representation of 
3 5 the object (the car) can be made well in advance, and only once 
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for a given car and a given advertisement, even if it is reused 
for different events . 

Then, the set of picture views and associated identi- 
fying parameters (PICTURE BASE) are downloaded (for example, 
5 broadcasted) at the beginning of the TV program. The identifying 
parameters take into account the moving object to which the 
picture has to be affected in case more than one moving object 
is processed by the system. The picture base is stored (block 
21, STP STORAGE) in the - W Set receiving the TV program and also 

10 'in the production or broadcast side (for example, in the signal 
distribution point 9, or in area distribution points 10) . 
Preferably, the picture base is stored in the set top box 11 of 
the TV set or of the digital decoder to which it is connected, 
which comprise calculation means. 

15 According to the invention, when an image is taken by 

a camera filming an event, the position and orientation of the 
camera (bloc 22, VTEW - LOCATION ORIENTATION) are calculated and 
transmitted along with the image to the production mixer. 

On the production (mixer) side, the position and 

2 0 orientation of the moving object (the car) is received through a 
positioning system. According to the invention, the moving 
objects which have to be processed to superimpose pictures are 
equipped not only with a positioning element but also with an 
orientation sensor. Indeed, to be able to determine the deforma- 

2 5 tion of the picture, the location coordinates of the objects are 

not sufficient. The orientation coordinates (yaw, pitch and 
roll) are also required. Miniaturized sensor of yaw, pitch and 
roll are commonly available and the information can be transmit- 
ted by the moving object along with its location coordinates. 

3 0 The same information are required for each camera so that the 

system is able, on the video production mixer, to compare the 
respective location and orientation of the object to those of 
the camera in order to determine the eventual changes in orien- 
tation and in size of the object in the current image with 
3 5 respect to the former one. 
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It is possible to deduce, from the location of the 
reference point of the object and its orientation parameters (in 
a reference system) , the eventual deformation of the pictures 
between the former and the current image. Further, knowing the 
5 difference of location of the reference point of the object 
between two successive images, it is easy to calculate the size 
variation of the picture between the two successive images. 

The detection of a new object appearing in a current 
image depends on the adaptation of the place of the event to the 
10 ' system of the invention. 

If the filmed place is not specifically adapted to the 
invention (for example, a car race on streets or a car rally) , 
the system can just use the location and orientation of the 
cameras and of the moving object to be processed. Then, the 
15 moving objects in the images are preferably detected with a 
shape recognition tool using geometrical representation of the 
obj ects . 

According to another embodiment, more particularly 
adapted to circumscribed event places (for example, car 

2 0 circuits); the place is provided with static localizable points 

regularly distributed on the place. Then, the supplemental 
information on the location of the moving objects with respect 
to these static points allows the detection of the apparition of 
a new object in a current image. 
25 Having determined the location and orientation of the 

pictures which should be present in the current image, the 
system (on the broadcast /production side) determines the 
oriented picture (by comparison with the set of preregistered 
oriented pictures) and the corresponding picture identifying 

3 0 parameters (comprising their respective orientations) and 

transmits, along with the image, the identifying parameter of 
the advertisement picture and the respective location and size 
information. 
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On the receiver (TV set) side, the identifying parame- 
ter is used to extract from a memory, the respective oriented 
picture (block 25, EXTRACT) . 

Then, the oriented picture is combined (superimposed) 
5 to the current video image in the right position (block 26, 
COMBINE) and the images are finally displayed (block 27, 
DISPLAY) . 

According to a first simplified embodiment where the 
signal distribution point 9 and area distribution points 10 are 

10 " mingled with the _ video picture mixer, the production/broadcast 
side calculates and transmits the complete information concern- 
ing the picture (i.e. the complete identifying parameters) . 

According to another embodiment, more particularly 
concerned with a system as represented in figure 2 in which the 

15 area distribution points are distinct from the point (video 
picture mixer 7 or signal distribution point 9) processing the 
steps 23/24, only the frame, i.e. orientation of the picture is 
transmitted without identifying the content thereof. Then, at 
the area distribution points 10, the content identifying refer- 

2 0 ence is combined with the information received concerning the 
shape, orientation and location of the picture. 

An advantage of the present invention is that the 
content of a picture on a moving object' can be adapted to the 
broadcasting area or other requirements. 

2 5 Another advantage of the present invention is that the 

superimposition of pictures is compatible with a real time 
calculation for live program. 

The practical implementation of the invention is in 
the ability of one with an ordinary skill in the art on the 

3 0 basis of the functional indications above. Especially, the 

moving objects usually carry more than one picture and the 
present disclosure made in connection with one picture extends 
to any number of pictures and any number of moving objects, 
provided that the pictures and the moving objects are locatable 
3 5 with respect to each other. 
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Figure 4 represents a block diagram of an exemplary 
embodiment of the present invention. Figure 4 outlines the 
differences with respect to a conventional system as shown in 
Figure 1. 

5 Instead of capturing the area where the event takes 

place, as in the existing method, the object - in our example a 
racecar - is filmed (block 50) from as many directions and pro- 
jections as necessary in order to project the advertising image 
on the required spot, ancl keep it there as if it was static. 

10 " These projections, are then stored in a main database 51 (shown 
to the left of the dashed line in Figure 4) , and also extracted 
(block 52) and transmitted in advance of the car race to the set 
top box 11 of the user to avoid repeat transmission of the same 
images. The transmission occurs, for example through a satellite 

15 8 (and earth base station 54 and receiver dish 55) Hence, the 
database shown as "magnified" (block 61) contains the necessary 
advertisement projections before the car race starts. 

When the race is running the advertisements will be 
inserted only in the designated spots on the car when nothing is 

2 0 blocking the camera view of the insertion spot. This is done in 
the same way as the previous method by comparing (block 56 or 
57) the stored camera view of the car with the current camera 
view selected by the mixer 9 among the camera views received 58, 
59, 60 from the various cameras. The position of the cameras and 

2 5 the car are obtained using GPS (assuming land-based radio 

support to improve accuracy) , and the orientation is obtained 
using accelerometers and/or gyroscopes inside each object. By 
combining this information a mathematical equation can express 
whether or not the current camera view has an advertisement spot 

3 0 or not . This does not take into account the previously mentioned 

obstructions that may be between the camera and the car, which 
therefore has to be tested manually by pixel comparisons . 

The only information thus needed while the race is 
running are an identifier of the advertisement, and an orienta- 
3 5 tion index to identify which projection of the advertisement 
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image should be placed on the car. Should any obstructions cover 
the advertisement spot wholly or partly, a polygon representa- 
tion can be calculated to mark the boundaries of the obstructing 
object, and sent along with the advertisement representation. 
5 When the advertisement representation is received by 

the viewer's set top box, indexes are used to look up the data- 
base 61 of advertisement image projections, and insert the 
appropriate image into the video stream to be watched on TV 62 . 
In the case of an obstruction, -the polygon representation of the 

10 ' boundaries of the obstructing object has to be translated to 
image pixel boundaries, and the part' of the advertisement that 
is obstructed will not be inserted in the video stream. 

The dashed box 57 in Figure 4 designates the broadcast 
to e.g. a different country, or area where e.g. tobacco and 

15 alcohol advertisements are illegal, or where different advertis- 
ing contracts have been signed. The same broadcast elements are 
used in this instance. However, they are left out here for space 
reasons. The difference is in the dashed box 57, which inserts 
and sends different advertisement images than the other 

2 0 broadcas t chain . 

A further possible method is to change the advertise- 
ment imagery during the race . Usually, more people watch the 
start and the end of a car race. As long as the images have the 
same boundaries, changing their pixel content is a minor issue, 

2 5 and this can be solved by e.g. downloading a double set of 

images to the set top box before the start of the race. 

Having thus described at least one illustrative 
embodiment of the invention, various alterations, modifications 
and improvements will readily occur to those skilled in the art. 

3 0 Such alteration, modification, and improvements are intended to 

be within the spirit and scope of the invention. Accordingly, 
the foregoing description is by way of example only and is not 
intended to be limiting. In particular, what has been disclosed 
in connection with car races can applies to other fields of 
3 5 application in which moving objects carrying advert pictures or 
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the like are filmed. Further, the implementation of the inven- 
tion with hardware/ software components is in the ability of 
those with an ordinary- skill in the art using known tolls. The 
invention is limited only as defined in the following claims and 
the equivalent thereto. 



